# Creating and Sharing Personal Web Archives

Tired of broken links? Archive yours with owark (http://owark.org), the Open Web Archive

Eric van der Vlist

OSCON 2011

Picture © harry harris http://www.flickr.com/photos/harryharris/2647935394/

# Disclaimer

This presentation is about a (crazy) idea as much as about an implementation

And the implementation is mainly a proof of concept

OK, but how do you present an idea?

Picture © archie2 http://www.flickr.com/photos/niekel/1682962952/

# Problem statement

> Link rot (or linkrot) ... is an informal term for the process by which, either on individual websites or the Internet in general, increasing numbers of links point to web pages, servers or other resources that have become permanently unavailable.

Source: http://en.wikipedia.org/wiki/Link_rot

Picture © ☮Stephanie http://www.flickr.com/photos/s_meshke/3893751728/

# An Old Issue

> 6% of the links on the Web are broken according to a recent survey by Terry Sullivan's All Things Web. Even worse, linkrot in May 1998 was double that found by a similar survey in August 1997.

Source: Jakob Nielsen's Alertbox for June 14, 1998

Picture © pellaea http://www.flickr.com/photos/7147684@N03/3260845004/

# That Should Never Happen

> Cool URIs don't change

Source: Tim Berners-Lee

Picture © Internet Society http://www.flickr.com/photos/internetsociety/5862727921/

# But Does Happen

> You can expect to lose about a quarter of them [ie outbound links] every seven years.

Source: Pinboard Blog

8% link rot within 1 year, 14% within 2 years, 28% within 3 years

Source: Breaking Down Link Rot: The Chesapeake Project Legal Information Archive's Examination of URL Stability

Picture © spablab http://www.flickr.com/photos/spablab/253756574/

# A Red Herring for some

> To be honest, I've rarely felt the need [to fight linkrot]. Content ages, by and large, and I inline quotes if I need a reference.

Picture © timparkinson http://www.flickr.com/photos/timparkinson/503768893/

# Links are a disruptive technology

> Links have become an essential part of how I write, and also part of how I read. Given a choice between reading something on paper and reading it online, I much prefer reading online: I can follow up on an article’s links to explore source material, gain a deeper understanding of a complex point, or just look up some term of art with which I’m unfamiliar.

Source: In Defense of Links, Part One: Nick Carr, hypertext and delinkification - Scott Rosenberg

Picture © danbri http://www.flickr.com/photos/danbri/4956640593/

# Links keep web pages alive

Web pages are more than text with added links.

Without their links web pages are dead!

Picture © Eric van der Vlist

# Tools against link rot (1/3)

## Link checkers

Browse websites to detect dead links

Page updates are manual or automatic (limited).

Picture © hugojcardoso http://www.flickr.com/photos/hugojcardoso/5939331519/

# Tools against link rot (2/3)

## Web archives

A daunting task for the internet at large (trillions of pages).

Picture © Laughing Squid http://www.flickr.com/photos/laughingsquid/4896726326/

# Tools against link rot (3/3)

## Local archives (caches)

Easier to set up.

In SGML/XML world, a good example are the Cover Pages.

Picture © Samuel Santos http://www.flickr.com/photos/samuel_santos/5445087726/

# Open Web Archive - owark: goal

Make it easy to integrate and deploy on your site:

- A link checker
- A local archive
- Site updates

Picture © Bogdan Suditu http://www.flickr.com/photos/bogdansuditu/2377844553/

# Open Web Archive - owark: plot

Try to sparkle multiple implementations sharing the same design

- In different languages
- Integrated into different web platforms

Share information between these implementations.

Picture © mirandaeostre http://www.flickr.com/photos/miranda_skye/4923830580/

# Is that legal?

I am not a lawyer, but...

Many sites already do that (think of Google cache...)

Good practices include:

- Adding a disclaimer when displaying archive pages
- Using a distinctive user agent string
- Supporting robots.txt

Picture © rkrichardson http://www.flickr.com/photos/45126397@N06/4990557451/

# Open Web Archive - owark: status

Still at a very early stage

A WordPress plugin is under development (in PHP)

Contributions are welcome

Picture © edgeplot http://www.flickr.com/photos/edgeplot/14877871/

# WordPress plugin (1/3)

## Link checker

Uses the Broken Link Checker plugin by Janis Elsts

Reads the table maintained by the BLC.

Picture © Calsidyrose http://www.flickr.com/photos/calsidyrose/5837350062/

# WordPress plugin (2/3)

## Page harvesting

Run as a WordPress cron job

Picture © Eric van der Vlist

# WordPress plugin (3/3)

## Archive display

Broken links are replaced by links to the local archive

A header is added to indicate that the page comes from an archive:

    This is an Open Web Archive archive of
    http://inventerre.canalblog.com/archives/2011/01/15/20122904.html.
    This snapshot has been taken on 2011-01-21 23:05:05 for the
    website Le retour à la terre which contains a link to this page
    and has saved a copy to be displayed in the page ever disappears.

Picture © Carbon Arc http://www.flickr.com/photos/41002268@N03/3879960518/

# WordPress plugin: lessons learned

Relatively easy to implement (~500 lines of PHP)

wget does most of the hard job

wget gets it right most of the time (80% ?)

wget has issues with Flash and tricky JavaScript

WordPress extension mechanisms more difficult to grok than the features to implement!

Picture © Photomatt28 http://www.flickr.com/photos/matthewpaulson/5948791784/

# WordPress plugin : next steps

Analyze pages that do not archive well and try to fix them

Better robots.txt support (robots.txt files can change!)

More user level setup.

Reimplementing the page harvesting in PHP (?)

Picture © Stuart Herbert http://www.flickr.com/photos/stuartherbert/4782800980/

# Sharing archive data

Toward a distributed Web archive...

If websites maintain local archives, why not share them?

Would leverage the web architecture to archive it

Picture © Chris-Håvard Berge http://www.flickr.com/photos/chberge/4670939397/

# A distributed Web archive, why?

The web is too big to fit into something centralized!

"Centralized" web archives need to be distributed over different servers

Distributing the archive over different organizations goes a step further

Picture © vshioshvili http://www.flickr.com/photos/vshioshvili/388221237/

# Bootstraping a distributed Web archive

Start simple: local archives would notify to a central server when they archive a page

The server would maintain a database

Archive data would be open and downloadable

A distributed database could be implemented as a second step

Picture © Al S http://www.flickr.com/photos/alandlesley/2113517073/

# Steps toward a distributed Web archive

Implement the database server (quite easy)

Update the WordPress plugin to send notifications (easy)

Create a community and get some traction (tough)...

... Other implementations are needed!

Picture © Chalky Lives http://www.flickr.com/photos/traitlinburke/2780059784/

# Help needed

If you're interested, you can help!

Discuss and document best practices for local archives

Improve the WordPress implementation

Write other implementations

Define and implement the database server

Install a local web archive on your server

Or just give feedback!

Picture © Easa Shamih (eEko) http://www.flickr.com/photos/eeko/5254059460/

# http://owark.org

A resource for the community

Currently includes trac and a git repository.

You're welcome to use it but can use whatever fits you better...

Ask me (vdv@dyomedea.com) if you want write access.

Picture © niallkennedy http://www.flickr.com/photos/niallkennedy/40727794/

# Questions? Feedback?

Picture © Seth Lemmons http://www.flickr.com/photos/slemmon/3957672912/